feat(stack): EQL v3 JSON support — types.Json + containment - #621
Conversation
…round-trip
PR4 (foundation). `types.Json('col')` declares a `public.eql_v3_json` column
that encrypts a JSON document to an ste_vec `SteVecDocument` and round-trips it
losslessly through encrypt/decrypt and the model path.
- columns.ts: add `'json'` to the cast_as kinds, a `JsonValue` plaintext type,
an `eql_v3_json` domain + `EncryptedJsonColumn` class, and a new OPTIONAL
`searchableJson` query capability that emits the `ste_vec` index (the scalar
3-flag model can't express it). `isQueryable`/`QueryableFlag` count it.
- The ste_vec index uses `mode: 'compat'`, which eql-3.0.0's `eql_v3_json`
REQUIRES: it orders entries by the CLLW-OPE `op` term, so v2's `'standard'`/
CLLW-`oc` terms are rejected at encrypt time. (Found by the round-trip test —
the FFI error names this exactly.)
- types.ts: `types.Json` factory (auto-registers in the domain-registry).
- test-kit: a deferred `eql_v3_json` catalog row (ste_vec containment doesn't
fit the scalar op oracle — covered by a dedicated suite), a `json` family for
coverage accounting, `searchableJson` in the op map, and the sample type
widened to include `JsonValue`.
- Updated the domain count/list assertions the 40th domain forces.
Verified against live crypto: the json round-trip suite passes (2/2). Unit +
type suites green (1676 + 93). Containment querying through the adapters is the
next increment.
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
`ops.contains(col, subObject)` on a `types.Json` column now answers
encrypted-JSONB containment. json has no `eql_v3.contains` overload, so
the operator emits `@>` with a `query_jsonb` needle built by
`encryptQuery` (no ciphertext), cast to `eql_v3.query_jsonb`.
The ste_vec index drops positional array terms (`array_index_mode`
keeps `item`+`wildcard`, not `position`) so containment is a true
jsonb subset test: `{ roles: ['x'] }` matches any document whose
`roles` array contains `x` regardless of index — verified live.
- src: sql-dialect `containsJson` (@> operator), operators `contains`
ste_vec branch + `encryptQuery` client capability, columns.ts index
- test: json-contains live suite (4), operators unit coverage (3)
- test-kit: catalog json row array-index-mode
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
- stash-encryption: types.Json in the capability-suffix and family tables, plus an "Encrypted-JSONB Queries" section (containment + JSONPath selector via encryptQuery searchableJson) - stash-drizzle: contains() answers JSON containment on a Json column (@> operator, query_jsonb needle, position-independent array subset) - changeset: stash patch (skills ship in the stash tarball) Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
🦋 Changeset detectedLatest commit: f4d45dd The changes in this PR will be included in the next version bump. This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Two CI failures, both from the JSON domain now living in the matrix:
- matrix-crypto round-trip asserted every encrypted field has a
top-level `c`. A JSON (`eql_v3_json`) value is an ste_vec document
(`{k:'sv', sv:[...]}`) with no top-level `c` — the ciphertext is
inside `sv`. Guard json by its `sv` array instead, mirroring the
json-crypto suite.
- The structural `OperandEncryptionClient.encryptQuery` pinned
`queryType: 'searchableJson'`, which the real client's generic
`encryptQuery` (queryType bound to the column's own query types)
could not satisfy, so `TypedEncryptionClient` stopped assigning.
Use `never` operands like `encrypt`; the call site casts.
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
The family-suite driver threw in planTable when a family had zero covered domains. The `json` family is exactly that by design — ste_vec containment/selector queries don't fit the scalar oracle, so its one domain is deferred and covered by dedicated suites (json-crypto, json-contains). runFamilySuite now emits a single assertion documenting the deferral for such a family (keeping it visible, not a silent skip), while a family with no domains at all still falls through to the throw as a wiring-bug guard. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
eql_v3_json is a modelled domain now (types.Json → in DOMAIN_REGISTRY),
so introspection correctly stops reporting a `doc eql_v3_json` column as
unmodelled. The suite predated json existing:
- the UNMODELLED table's `doc` is now a modelled column (synthesized,
columnBuilders=['doc']); only `score` (eql_v3_integer_ord_ope, still
no factory) is reported unmodelled
- the factory sanity check keyed by BARE names ('json') which never
match the registry's `eql_v3_*` keys — a latent no-op. Key it by the
real `eql_v3_*` domain name so it actually asserts json is modelled
and integer_ord_ope is not
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
Test-coverage review — EQL v3 JSON (ste_vec) containment
Strong coverage overall: the built column shape is pinned in v3-matrix/matrix.test.ts (build() toStrictEqual over the catalog), the registry key in eql-v3-domain-registry.test.ts, the family partition/deferral in test-kit-families.test.ts, plus live round-trip (json-crypto) and live containment incl. position-independent array matching (json-contains, ada matches with eng at index 1). The unit suite covers the happy @> path, encryptQuery failure, and null-operand rejection.
Two gaps stand out, both mirroring anti-patterns already handled elsewhere in this file:
- The new
encryptJsonContainmentTermguard for a client withoutencryptQueryhas no test — thesetup()double always supplies it, so the branch the code comments call out as load-bearing is never exercised. (inline) - Empty-object containment (
contains(col, {})) is unpinned. The bloom path deliberately keepsrequireAnswerableNeedlebecause an empty needle makes@> '{}'match every row; the ste_vec path added here has no equivalent guard and no test documenting the behaviour. This is a lopsided-negative-case gap with a whole-table-leak flavour — flagging it as a coverage gap, not asserting a bug. (inline)
Additional coverage gaps not posted inline
- Top-level non-object JSON round-trip.
JsonValue(andtypes.Json) admit a top-level array, scalar, ornull, butjson-crypto.integration.test.tsonly round-trips objects. A regression in ste_vec encoding of non-object roots would go unnoticed. Add a case round-tripping e.g.[1, 2, 3]andnull(or, if only object roots are supported, assert the rejection so the boundary is explicit).
There was a problem hiding this comment.
Pull request overview
Adds first-class EQL v3 encrypted JSON support to @cipherstash/stack via a new typed-schema column factory (types.Json) backed by an ste_vec index, plus Drizzle-side JSON containment querying that emits @> with a narrowed query_jsonb needle from encryptQuery. Also updates the published skills docs and expands the shared integration test-kit catalog to include (deferred) JSON domains while covering behavior with dedicated live suites.
Changes:
- Add
types.Json('col')/EncryptedJsonColumnandJsonValuetyping, including ste_vec index emission (mode: 'compat', positionless array containment). - Implement Drizzle v3 JSON containment dispatch in
contains()to emitOPERATOR(public.@>) …::eql_v3.query_jsonbusingencryptQuery. - Add/adjust dedicated integration + unit tests, plus update shipped
skills/*/SKILL.mdand include changesets for stack + stash (skills).
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| skills/stash-encryption/SKILL.md | Documents types.Json in the v3 typed schema catalog and adds encrypted-JSONB query guidance. |
| skills/stash-drizzle/SKILL.md | Documents that contains supports JSON containment for types.Json columns. |
| packages/test-kit/src/run-family-suite.ts | Handles fully-deferred families (like json) without silently skipping. |
| packages/test-kit/src/oracle.ts | Narrows widened samples back to scalar Plain for the scalar oracle. |
| packages/test-kit/src/ops.ts | Adds searchableJson capability key mapping (for type completeness). |
| packages/test-kit/src/families.ts | Adds json as a family and marks it deferred for coverage accounting. |
| packages/test-kit/src/catalog.ts | Adds the public.eql_v3_json domain spec (deferred) with ste_vec index config and samples. |
| packages/stack/src/eql/v3/types.ts | Adds types.Json factory to construct EncryptedJsonColumn. |
| packages/stack/src/eql/v3/index.ts | Re-exports EncryptedJsonColumn and JsonValue. |
| packages/stack/src/eql/v3/drizzle/sql-dialect.ts | Adds containsJson dialect helper emitting @> with query_jsonb cast. |
| packages/stack/src/eql/v3/drizzle/operators.ts | Routes contains() to JSON containment when ste_vec index is present; uses encryptQuery for the needle. |
| packages/stack/src/eql/v3/columns.ts | Adds searchableJson capability plumbing, ste_vec index defaults, EncryptedJsonColumn, and JsonValue plaintext mapping. |
| packages/stack/integration/supabase/introspect.integration.test.ts | Updates introspection expectations now that eql_v3_json is modelled by types.Json. |
| packages/stack/integration/shared/matrix-crypto.integration.test.ts | Adjusts ciphertext-shape guard to handle ste_vec (k:'sv') payloads. |
| packages/stack/integration/shared/json-crypto.integration.test.ts | New live round-trip suite proving JSON encrypt/decrypt and model-path round-trip. |
| packages/stack/integration/drizzle-v3/json-contains.integration.test.ts | New live Drizzle containment suite for encrypted JSONB (ops.contains on types.Json). |
| packages/stack/tests/test-kit-families.test.ts | Updates deferred-family expectations to include json. |
| packages/stack/tests/supabase-v3-matrix.test.ts | Updates domain tier counts to include eql_v3_json. |
| packages/stack/tests/eql-v3-domain-registry.test.ts | Pins the new eql_v3_json registry key. |
| packages/stack/tests/drizzle-v3/operators.test.ts | Adds unit assertions for JSON containment SQL emission and error surfacing. |
| .changeset/eql-v3-json.md | Changeset for @cipherstash/stack minor release describing v3 JSON support. |
| .changeset/eql-v3-json-skills.md | Changeset for stash patch release due to shipped skill updates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Code review — EQL v3 JSON support (
|
Review feedback on PR #621 (coderdan test-gaps + Copilot): - Empty-object containment leak: `contains(col, {})` matched every row (`doc @> '{}'` holds for all), the whole-table footgun the bloom path already guards. Reject an empty-object needle in the ste_vec path with a typed EncryptionOperatorError; pin it in both the unit and live suites. - Type/runtime mismatch: `types.Json` plaintext was `JsonValue` (admits top-level scalars), but protect-ffi rejects a bare scalar ("Cannot convert … to Json"). Narrow the column plaintext to a new `JsonDocument` (object | array | null); keep `JsonValue` for nested values. Export it. - encryptQuery-absent guard was untested (the double always supplied it). Add a unit test with a `{ encrypt }`-only client. - Non-object roots unpinned: add live round-trips for array and null roots, and assert the top-level-scalar rejection so the boundary is explicit. - Copilot: json-contains header said `eql_v3.contains`; it's the `@>` operator. Corrected. Skill updated to describe JsonDocument. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
Review findings addressed — 58b9ef8Thanks both. All four addressed (none refuted — each was a real gap or a real defect): 1. 2. Empty-object containment (coderdan) — probed live: 3. Non-object JSON roots (coderdan) — the finding surfaced a genuine type/runtime mismatch. Live behaviour: object, array, and 4. Misleading header comment (Copilot) — corrected: the New/changed public surface: |
Review triage — responses in-threadReplied to each thread above. Summary of dispositions: Follow-up issues (out of scope for this PR, by design — keeping it to column + round-trip + containment):
Wording / hygiene to fix in this PR (once you give the go-ahead):
Parked: whether EQL v3 should have a storage-only (no-query) JSON domain at all. Today there's exactly one JSON domain, |
…nge) Addresses the PR #621 review's wording/classification/test-hygiene points (design changes split to #622/#623): - Sweep `ste_vec` out of user-facing comments/test names/skills → "encrypted JSONB document" / `eql_v3_json`; keep `ste_vec` only where it is literally the protect-ffi config key (`indexes.ste_vec`) or the `SteVecDocument` payload type. - json-contains suite: drop the `makeEqlV3Column` wrapper (use the Drizzle `types.Json`), which lets both `as never` casts go (the pgTable and the bulkEncryptModels call) — the table is now properly typed; un-alias the Drizzle `eq`/`asc` imports. - operators unit: rename `storageDomains` → `nonScalarQueryDomains` and fix the comment — json is a QUERYABLE (containment) domain that answers no scalar op, not a storage-only one. - Reframe json's `deferred` marker across catalog/families/run-family-suite/ test: it means "not run by the scalar op-matrix" (covered by dedicated suites), NOT unimplemented — distinct from the ORE domains' superuser-only deferral. - Rewrite the opaque introspect comment as the explicit three-way case. - Skill: describe JSON as containment-only today; mark JSONPath selector as not-yet-implemented (tracked in #623); note `JsonDocument` (no top-level scalar). Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
Wording / hygiene batch landed — e27ce9dThe non-design review points are addressed (design ones split to #622/#623):
Scope note added to the PR description. All green locally: unit 1680, type tests 93, json-contains 5, json-crypto 7, introspect 6. |
…into feat/eql-v3-json # Conflicts: # packages/stack/src/eql/v3/drizzle/operators.ts
coderdan
changed the title
EQL v3 JSON support (part 1)
Adds encrypted JSON columns to the EQL v3 typed schema:
types.Json('col'), alossless round-trip, and Drizzle containment querying. This is the last PR in the
integration-tests → type-robustness → live-suite-reorg → JSON sequence.
Scope: containment only
This PR implements JSON containment (
ops.contains(col, subObject)→@>). Two JSON concerns are deliberately out of scope, tracked as follow-ups:metadata->'plan' = $1) — theeql_v3_jsondomain supports->/->>at the SQL level, but the Drizzle operator and the selector-string needle typing are not wired up. → EQL v3 JSON: implement JSONPath selector-with-constraint querying (->, ->>) #623encryptQuery(today only JSON containment uses it; scalar ops still useclient.encrypt). → EQL v3 Drizzle: encrypt all query operands with encryptQuery (not client.encrypt) #622Also noted for the EQL team: v3 exposes only one JSON domain (
eql_v3_json), which requires the ste_vec document structure — there is no storage-only / no-query JSON domain equivalent to v2'sdataType('json')without.searchableJson().What's new
types.Json('metadata')declares apublic.eql_v3_jsoncolumn. It encrypts awhole JSON document to an ste_vec
SteVecDocumentand round-trips losslesslythrough
encrypt/decryptand the model path. The newsearchableJsonquerycapability emits the ste_vec index.
Drizzle containment —
ops.contains(col, subObject)now answersencrypted-JSONB containment on a
Jsoncolumn, returning exactly the rows whosedocument contains the sub-object (jsonb
@>semantics).Two findings worth calling out
eql_v3.containsoverload. Containment is the@>operator —public.@>(eql_v3_json, eql_v3.query_jsonb). The dialect emitscol OPERATOR(public.@>) needle::eql_v3.query_jsonb, where the needle is anarrowed
query_jsonbterm fromencryptQuery(no ciphertext), not the fullstorage envelope. This is what resolves the "function is not unique" (42725)
ambiguity.
searchableJson()usesarray_index_mode: 'all', which emits a positional termper element — so
{roles:['eng']}matched a doc withengat index 0 but notone with
engat index 1. TheJsoncolumn keepsitem+wildcardand dropsposition, making containment a true subset test regardless of element order.That is the correct default for a containment column.
mode: 'compat'is requiredeql_v3_jsonorders ste_vec entries by the CLLW-OPEopterm, so the index mustemit compat terms. v2's
'standard'(CLLW-OREoc) is rejected at encrypt time.Naming note (
ste_vecvseql_v3_json)EQL 3.0.0 renamed the column/domain types (
public.eql_v3_json,eql_v3_jsonb_entry— there is noste_vectype). Butste_vecremains the nameof the index kind in the protect-ffi config, the payload shape
(
SteVecDocument,k:"sv"), and internal SQL functions. Allste_vecreferencesin the code sit in exactly those layers; nothing user-facing uses it.
Tests & verification (live pg)
integration/shared/json-crypto— round-trip, scalar + model paths (2/2)integration/drizzle-v3/json-contains— object / scalar / nested-boolean /array-subset / no-match containment (4/4)
operators.test— json@>SQL emission, encryptQuery-failure surfacing,null-operand rejection (3 new); full stack unit suite 1650 passing
Skills
Per the same-PR rule:
stash-encryptiongainstypes.Jsonin its typed-schemacatalog and an encrypted-JSONB query section;
stash-drizzledocuments jsoncontainment on
contains. Carries astashpatch changeset (skills ship in thetarball) alongside the
@cipherstash/stackminor.https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w